The present invention relates to apparatus and methods for monitoring environments in and outside of a vehicle using image processing.
The present invention also relates to arrangements for detecting the presence, type and/or position of occupants in vehicles and objects exterior of vehicles, e.g., in a driver""s blind spot, primarily using optics.
The present invention also relates to apparatus and methods for determining a distance between objects in an environment in and outside of a vehicle by image processing techniques.
1. Prior Art on Out of Position Occupants and Rear Facing Child Seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and over 100 people have now been killed. Thus, significant improvements need to be made to airbag systems. As discussed in detail in U.S. Pat. No. 5,653,462 referenced above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. xe2x80x9cHow Airbags Workxe2x80x9d presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection systems are required.
Initially, these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system will therefore be needed to detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782 will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. As discussed more fully below, the instant invention is an improvement on that VIMS system and uses an advanced optical system comprising one or more CCD (charge coupled device) or CMOS arrays and particularly active pixel arrays plus a source of illumination preferably combined with a trained neural network pattern recognition system.
Others have observed the need for an occupant out-of-position sensor and several methods have been disclosed in U.S. patents for determining the position of an occupant of a motor vehicle. Each of these systems, however, has significant limitations. For example, in White et al. (U.S. Pat. No. 5,071,160), a single acoustic sensor and detector is described and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. The sensors measure the change in position of an occupant during a crash and use that information to access the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in one or more of the above-referenced patents and patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use active or passive infrared to determine the position of the occupant. As pointed out in one or more of the above-referenced patents and patent applications, direct occupant position measurement based on passive infrared is probably not possible and, until very recently, was very difficult and expensive with active infrared requiring the modulation of an expensive GaAs infrared laser. Since there is no mention of these problems, the method of use contemplated by Mattes et al. must be similar to the electric eye concept where position is measured indirectly as the occupant passes by a plurality of longitudinally spaced-apart sensors.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. describe only lower mounting locations of their sensors in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant""s hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver""s arms are the closest objects to two of the sensors.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant""s hands as in the case where he/she is adjusting the knob on the radio and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant""s head or chest. Since the sizes and driving positions of occupants are extremely varied, it is now believed that pattern recognition systems and preferably trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position. In fact, the closer the occupant gets to the airbag, the faster the inflation rate of the airbag is according to the Fujita et al. patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat (occupant height). This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant""s head.
As discussed above, the optical systems described herein are also applicable for many other sensing applications both inside and outside of the vehicle compartment such as for sensing crashes before they occur as described in U.S. Pat. No. 5,829,782, for a smart headlight adjustment system and for a blind spot monitor (also disclosed in U.S. provisional patent application Ser. No. 60/202,424).
2. Definitions
Preferred embodiments of the invention are described below and unless specifically noted, it is the applicants"" intention that the words and phrases in the specification and claims be given the ordinary and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicant intends any other meaning, he will specifically state he is applying a special meaning to a word or phrase.
Likewise, applicants"" use of the word xe2x80x9cfunctionxe2x80x9d here is not intended to indicate that the applicants seek to invoke the special provisions of 35 U.S.C. xc2xa7112, sixth paragraph, to define their invention. To the contrary, if applicants wish to invoke the provisions of 35 U.S.C. xc2xa7112, sixth paragraph, to define their invention, they will specifically set forth in the claims the phrases xe2x80x9cmeans forxe2x80x9d or xe2x80x9cstep forxe2x80x9d and a function, without also reciting in that phrase any structure, material or act in support of the function. Moreover, even if applicants invoke the provisions of 35 U.S.C. xc2xa7112, sixth paragraph, to define their invention, it is the applicants"" intention that their inventions not be limited to the specific structure, material or acts that are described in the preferred embodiments herein. Rather, if applicants claim their inventions by specifically invoking the provisions of 35 U.S.C. xc2xa7112, sixth paragraph, it is nonetheless their intention to cover and include any and all structure, materials or acts that perform the claimed function, along with any and all known or later developed equivalent structures, materials or acts for performing the claimed function.
The use of pattern recognition is important to the instant invention as well as to one or more of those disclosed in the above-referenced patents and patent applications above. xe2x80x9cPattern recognitionxe2x80x9d as used herein will generally mean any system which processes a signal that is generated by an object, or is modified by interacting with an object, in order to determine which one of a set of classes that the object belongs to. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally electrical signals coming from transducers which are sensitive to either acoustic or electromagnetic radiation and, if electromagnetic, they can be either visible light, infrared, ultraviolet or radar or low frequency radiation as used in capacitive sensing systems.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught various patterns by subjecting the system to a variety of examples. The most successful such system is the neural network. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, sensor fusion, Kalman filters, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
A pattern recognition algorithm will thus generally mean an algorithm applying or obtained using any type of pattern recognition system, e.g., a neural network, sensor fusion, fuzzy logic, etc.
To xe2x80x9cidentifyxe2x80x9d as used herein will usually mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
To xe2x80x9cascertain the identity ofxe2x80x9d as used herein with reference to an object will generally mean to determine the type or nature of the object (obtain information as to what the object is), i.e., that the object is an adult, an occupied rear facing child seat, an occupied front facing child seat, an unoccupied rear facing child seat, an unoccupied front facing child seat, a child, a dog, a bag of groceries, a car, a truck, a tree, a pedestrian, a deer etc.
An xe2x80x9coccupying itemxe2x80x9d or xe2x80x9coccupantxe2x80x9d of a seat or xe2x80x9cobjectxe2x80x9d in a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
A xe2x80x9crear seatxe2x80x9d of a vehicle as used herein will generally mean any seat behind the front seat on which a driver sits. Thus, in minivans or other large vehicles where there are more than two rows of seats, each row of seats behind the driver is considered a rear seat and thus there may be more than one xe2x80x9crear seatxe2x80x9d in such vehicles. The space behind the front seat includes any number of such rear seats as well as any trunk spaces or other rear areas such as are present in station wagons.
An optical image will generally mean any type of image obtained using electromagnetic radiation including visual, infrared and radar radiation.
In the description herein on anticipatory sensing, the term xe2x80x9capproachingxe2x80x9d when used in connection with the mention of an object or vehicle approaching another will usually mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The xe2x80x9ctargetxe2x80x9d vehicle is the vehicle that is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
xe2x80x9cOut-of-positionxe2x80x9d as used for an occupant will generally mean that the occupant, either the driver or a passenger, is sufficiently close to an occupant protection apparatus (airbag) prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. It may also mean that the occupant is not positioned appropriately in order to attain the beneficial, restraining effects of the deployment of the airbag. As for the occupant being too close to the airbag, this typically occurs when the occupant""s head or chest is closer than some distance such as about 5 inches from the deployment door of the airbag module. The actual distance where airbag deployment should be suppressed depends on the design of the airbag module and is typically farther for the passenger airbag than for the driver airbag.
3. Pattern Recognition Prior Art
Japanese Patent No. 3-42337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and analyzes only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver""s mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates on locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver""s eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of illumination of the occupant, monitoring the position of the occupant, other that the eyes, determining the position of the eyes relative to the passenger compartment, or identifying any other object in the vehicle other than the driver""s eyes. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant utilizing two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver""s eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. A review of this book did not appear to provide the answer to this question. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with active infrared radiation or in general electromagnetic radiation.
The use of neural networks as the pattern recognition technology is important to several of the implementations of this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few hundred lines of code written in the C or C++ computer language as opposed to typically many hundreds of lines when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been prohibitively expensive until recently, rendering their use for VIMS impractical. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities.
The present invention preferably uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. xe2x80x9cAnalysis of Hidden Units in a Layered Network Trained to Classify Sonar Targetsxe2x80x9d, Neural Networks, Vol. 1. pp. 75-89, 1988, and xe2x80x9cLearned Classification of Sonar Targets Using a Massively Parallel Networkxe2x80x9d, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled xe2x80x9cProcess and Apparatus for the Automatic Detection and Extraction of Features in Images and Displaysxe2x80x9d to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled xe2x80x9cArtificial Neuron and Method of Using Same and U.S. Pat. No. 5,517,667 entitled xe2x80x9cNeural Network and Method of Using Samexe2x80x9d to S. T. Wang. Other examples include U.S. Pat. Nos. 5,235,339 (Morrison et al.), 5,214,744 (Schweizer et al), 5,181,254 (Schweizer et al), and 4,881,270 (Knecht et al). All of the above references are incorporated herein by reference. Neural networks as used herein include all types of neural networks including modular neural networks, cellular neural networks and support vector machines and all combinations as described in detail in U.S. Pat. No. 6,445,988, which is incorporated herein by reference in its entirety, and referred to therein as xe2x80x9ccombination neural networksxe2x80x9d.
4. Optics
Optics can be used in several configurations for monitoring the interior of a passenger compartment or exterior environment of an automobile. In one known method, a laser optical system uses a GaAs infrared laser beam to momentarily illuminate an object, occupant or child seat, in the manner as described and illustrated in FIG. 8 of U.S. Pat. No. 5,829,782 referenced above. The receiver can be a charge-coupled device or CCD (a type of TV camera), or a CMOS imager to receive the reflected light. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest within or around the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in the commonly owned ""462 patent as also described below. When a single receiving element is used, a PIN or avalanche diode is preferred.
In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to illuminate the desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. Recently, however, the cost of CCD and CMOS arrays has dropped substantially with the result that this configuration may now be the most cost-effective system for monitoring the passenger compartment as long as the distance from the transmitter to the objects is not needed. If this distance is required, then the laser system, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein is required. Alternately, a modulation system such as used with the laser distance system can be used with a CCD or CMOS camera and distance determined on a pixel by pixel basis.
A mechanical focusing system, such as used on some camera systems can determine the initial position of an occupant but is too slow to monitor his/her position during a crash. Although the example of an occupant is used herein as an example, the same or similar principles apply to objects exterior to the vehicle. A distance measuring system based on focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras, the latter of which would be fast enough. Although the Subbarao patent provides a good discussion of the camera focusing art and is therefore incorporated herein by reference, it is a more complicated system than is needed for the practicing the instant invention. In fact, a neural network can also be trained to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD""s and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of this instant invention. Distance can also be determined by the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by the spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can also be measured by using time of flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching of this invention.
In each of these cases, regardless of the distance measurement system used; a trained pattern recognition system, as defined above, is used in the instant invention to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
5. Optics and Acoustics
The laser systems described above are expensive due to the requirement that they be modulated at a high frequency if the distance from the airbag to the occupant, for example, needs to be measured. Alternately, modulation of another light source such as an LED can be done and the distance measurement accomplished using a CCD or CMOS array on a pixel by pixel basis.
Both laser and non-laser optical systems in general are good at determining the location of objects within the two dimensional plane of the image and a pulsed laser radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time of flight through range gating techniques. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements as discussed below.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z axis is from the transducer to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. Nos. 5,845,000 and 5,835,613.
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and CCD or CMOS array which receives and displays the reflected light and an analog to digital converter (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to a feature extraction algorithm (such as an edge detector) and then to an Artificial Neural Network (ANN) or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the driver""s ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant""s ears.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the occupant can be updated to approximately 10 milliseconds, which though sufficient for most cases, is marginal if the position of the occupant is to be tracked during a vehicle crash. Second, ultrasound waves are diffracted by changes in air density that can occur when the heater or air conditioner is operated or when there is a high-speed flow of air past the transducer. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
6. Applications
The applications for this technology are numerous as described in the patents and patent applications listed above. They include: (i) the monitoring of the occupant for safety purposes to prevent airbag deployment induced injuries, (ii) the locating of the eyes of the occupant (driver) to permit automatic adjustment of the rear view mirror(s), (iii) the location of the seat to place the occupant""s eyes at the proper position to eliminate the parallax in a heads-up display in night vision systems, (iv) the location of the ears of the occupant for optimum adjustment of the entertainment system, (v) the identification of the occupant for security reasons, (vi) the determination of obstructions in the path of a closing door or window, (vii) the determination of the position of the occupant""s shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant, (viii) the determination of the position of the rear of the occupants head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts, (ix) anticipatory crash sensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii) sunlight and headlight glare reduction and many others. In fact, over forty products alone have been identified based on the ability to identify and monitor objects and parts thereof in the passenger compartment of an automobile or truck. In addition, there are many applications of the apparatus and methods described herein for monitoring the environment exterior to the vehicle.
7. Other Prior Art
European Patent Application No. 98110617.2 (Publication No. 0 885 782 A1), corresponding to U.S. patent application Ser. No. 08/872,836 filed Jun. 11, 1997, describes a purportedly novel motor vehicle control system including a pair of cameras which operatively produce first and second images of a passenger area. A distance processor determines the distances that a plurality of features in the first and second images are from the cameras based on the amount that each feature is shifted between the first and second images. An analyzer processes the determined distances and determines the size of an object on the seat. Additional analysis of the distance also may determine movement of the object and the rate of movement. The distance information also can be used to recognize predefined patterns in the images and this identify objects. An air bag controller utilizes the determined object characteristics in controlling deployment of the air bag.
A paper entitled xe2x80x9cSensing Automobile Occupant Position with Optical Triangulationxe2x80x9d by W. Chappelle, Sensors, December 1995, describes the use of optical triangulation techniques for determining the presence and position of people or rear-facing infant seats in the passenger compartment of a vehicle in order to guarantee the safe deployment of an air bag. The paper describes a system called the xe2x80x9cTakata Safety Shieldxe2x80x9d which purportedly makes high-speed distance measurements from the point of air bag deployment using a modulated infrared beam projected from an LED source. Two detectors are provided, each consisting of an imaging lens and a position-sensing detector.
A paper entitled xe2x80x9cAn Interior Compartment Protection System based on Motion Detection Using CMOS Imagersxe2x80x9d by S. B. Park et al., 1998 IEEE International Conference on Intelligent Vehicles, describes a purportedly novel image processing system based on a CMOS image sensor installed at the car roof for interior compartment monitoring including theft prevention and object recognition. One disclosed camera system is based on a CMOS image sensor and a near infrared (NIR) light emitting diode (LED) array.
A paper entitled xe2x80x9cA 256xc3x97256 CMOS Brightness Adaptive Imaging Array with Column-Parallel Digital Outputxe2x80x9d by C. Sodini et al., 1988 IEEE International Conference on Intelligent Vehicles, describes a CMOS image sensor for intelligent transportation system applications such as adaptive cruise control and traffic monitoring. Among the purported novelties is the use of a technique for increasing the dynamic range in a CMOS imager by a factor of approximately 20, which technique is based on a previously described technique for CCD imagers.
A paper entitled xe2x80x9cIntelligent System for Video Monitoring of Vehicle Cockpitxe2x80x9d by S. Boverie et al., SAE Technical Paper Series No. 980613, Feb. 23-26, 1998, describes the installation of an optical/retina sensor in the vehicle and several uses of this sensor. Possible uses are said to include observation of the driver""s face (eyelid movement) and the driver""s attitude to allow analysis of the driver""s vigilance level and warn him/her about critical situations and observation of the front passenger seat to allow the determination of the presence of somebody or something located on the seat and to value the volumetric occupancy of the passenger for the purpose of optimizing the operating conditions for air bags.
Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an image analyzer (M5 in FIG. 1) for analyzing the position of driver including an infrared light source which illuminates the driver""s face and an image detector which receives light from the driver""s face, determines the position of facial feature, e.g., the eyes in three dimensions, and thus determines the position of the driver in three dimensions. A pattern recognition process is used to determine the position of the facial features and entails converting the pixels forming the image to either black or white based on intensity and conducting an analysis based on the white area in order to find the largest contiguous white area and the center point thereof Based on the location of the center point of the largest contiguous white area, the driver""s height is derived and a heads up display is adjusted so information is within driver""s field of view. The pattern recognition process can be applied to detect the eyes, mouth, or nose of the driver based on the differentiation between the white and black areas.
Ando (U.S. Pat. No. 5,008,946) describes a system which recognizes an image and specifically ascertains the position of the pupils and mouth of the occupant to enable movement of the pupils and mouth to control electrical devices installed in the automobile. The system includes a camera which takes a picture of the occupant and applies algorithms based on pattern recognition techniques to analyze the picture, converted into an electrical signal, to determine the position of certain portions of the image, namely the pupils and mouth.
Masamori (U.S. Pat. No. 5,227,784) describes a system which is based on radar, specifically it is a collision avoidance system aimed at detecting vehicles which are at some distance from the vehicle.
Suzuki et al. (U.S. Pat. No. 5,026,153) describes a vehicle tracking control for continuously detecting the distance and direction to a preceding vehicle irrespective of background dark/light distribution. In this system, every vehicle must have a light on its rear that emits a constant or time varying signal and two photoelectric sensors that zero in on the light emitted from the preceding vehicle are used and thereby determine both the distance and angular position of the preceding vehicle.
Krumm (U.S. Pat. No. 5,983,147) describes a system for determining the occupancy of a passenger compartment including a pair of cameras mounted so as to obtain binocular stereo images of the same location in the passenger compartment. A representation of the output from the cameras is compared to stored representations of known occupants and occupancy situations to determine which stored representation the output from the cameras most closely approximates. The stored representations include that of the presence or absence of a person or an infant seat in the front passenger seat.
Farmer et al. (U.S. Pat. No. 6,005,958) describes a method and system for detecting the type and position of a vehicle occupant utilizing a single camera unit. The single camera unit is positioned at the driver or passenger side A-pillar in order to generate data of the front seating area of the vehicle. The type and position of the occupant is used to optimize the efficiency and safety in controlling deployment of an occupant protection device such as an air bag.
A paper by Rudolf Schwarte, et al. entitled xe2x80x9cNew Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technologyxe2x80x9d, Eds. S. Krueger, W. Gessner, Proceedings of the AMAA 2000 Advanced Microsystems for Automotive Applications 2000, Springer Verlag; Berlin, Heidelberg, N.Y., ISBN 3-540-67087-4, describes an implementation of the teachings of the instant invention wherein a modulated light source is used in conjunction with phase determination circuitry to locate the distance to objects in the image on a pixel by pixel basis. This camera is an active pixel camera the use of which for internal and external vehicle monitoring is also a teaching of this invention. The novel feature of the PMD camera is that the pixels are designed to provide a distance measuring capability within each pixel itself. This then is a novel application of the active pixel and distance measuring teachings of the instant invention.
The instant invention as described in the above-referenced commonly assigned patents and patent applications, teaches the use of modulating the light used to illuminate an object and to determine the distance to that object based on the phase difference between the reflected radiation and the transmitted radiation. The illumination can be modulated at a single frequency when short distances such as within the passenger compartment are to be measured. Typically, the modulation wavelength would be selected such that one wave would have a length of approximately one meter or less. This would provide resolution of 1 cm or less. For larger vehicles, a longer wavelength would be desirable. For measuring longer distances, the illumination can be modulated at more than one frequency to eliminate cycle ambiguity if there is more than one cycle between the source of illumination and the illuminated object. This technique is particularly desirable when monitoring objects exterior to the vehicle to permit accurate measurements of devices that are hundreds of meters from the vehicle as well as those that are a few meters away. Naturally, there are other modulation methods that eliminate the cycle ambiguity such as modulation with a code that is used with a correlation function to determine the phase shift or time delay. This code can be a pseudo random number in order to permit the unambiguous monitoring of the vehicle exterior in the presence of other vehicles with the same system. This is sometimes known as noise radar, noise modulation (either of optical or radar signals), ultra wideband (UWB) or the techniques used in Micropower impulse radar (MIR).
Although a simple frequency modulation scheme has been disclosed so far, it is also possible to use other coding techniques including the coding of the illumination with one of a variety of correlation patterns including a pseudo-random code. Similarly, although frequency and code domain systems have been described, time domain systems are also applicable wherein a pulse of light is emitted and the time of flight measured. Additionally, in the frequency domain case, a chirp can be emitted and the reflected light compared in frequency with the chirp to determine by frequency difference the distance to the object. Although each of these techniques is known to those skilled in the art, they have heretofore not been applied for monitoring objects within or outside of a vehicle.
Principle objects and advantages of the optical sensing system in accordance with the invention are:
1. To provide a system for monitoring the environment exterior of a vehicle in order to determine the presence and classification, identification and/or location of objects in the exterior environment.
2. To provide a blind spot detector which detects and categorizes an object in the driver""s blind spot or other location in the vicinity of the vehicle, and warns the driver in the event the driver begins to change lanes, for example, or continuously informs the driver of the state of occupancy of the blind spot.
3. To provide a camera system for interior and exterior monitoring, which can adjust on a pixel by pixel basis for the intensity of the received light.
4. To provide for the use of an active pixel camera for interior and exterior vehicle monitoring.
5. To recognize the presence of a human on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag, heating and air conditioning, or entertainment systems, among others.
6. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her position and to use this position information to affect the operation of another vehicle system.
7. To determine the position, velocity or size of an occupant in a motor vehicle and to utilize this information to control the rate of gas generation, or the amount of gas generated by an airbag inflator system.
8. To determine the presence or position of rear seated occupants in the vehicle and to use this information to affect the operation of a rear seat protection airbag for frontal, side and/or rear impacts.
9. To recognize the presence of a rear facing child seat on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system.
10. To determine the approximate location of the eyes of a driver and to use that information to control the position of one or more of the rear view mirrors of the vehicle.
11. To monitor the position of the head of the vehicle driver and determine whether the driver is falling asleep or otherwise impaired and likely to lose control of the vehicle and to use that information to affect another vehicle system.
12. To provide an occupant position sensor which reliably permits, and in a timely manner, a determination to be made that the occupant is out-of-position, or will become out-of-position, and likely to be injured by a deploying airbag and to then output a signal to suppress the deployment of the airbag.
13. To provide an anticipatory sensor that permits accurate identification of the about-to-impact object in the presence of snow and/or fog whereby the sensor is located within the vehicle.
14. To provide a smart headlight dimmer system which senses the headlights from an oncoming vehicle or the tail lights of a vehicle in front of the subject vehicle and identifies these lights differentiating them from reflections from signs or the road surface and then sends a signal to dim the headlights.
15. To provide an occupant position determination in a sufficiently short time that the position of an occupant can be tracked during a vehicle crash.
16. To provide an occupant vehicle interior monitoring system which is not affected by temperature or thermal gradients.
17. To provide an occupant vehicle interior monitoring system which has high resolution to improve system accuracy and permits the location of body parts of the occupant to be determined.
18. To provide an occupant vehicle interior monitoring system which reduces the glare from sunlight and headlights by imposing a filter between the eyes of an occupant and the light source.
19. To provide a system for recognizing the identity of a particular individual in the vehicle.
20. To use the principles of time of flight to measure the distance to an occupant or object exterior to the vehicle.
21. To obtain a three dimensional image from a device at one location on a vehicle.
22. To use pattern recognition techniques for analyzing three-dimensional image data of occupants of a vehicle and objects exterior to the vehicle.
23. To provide a system of frequency domain modulation of the illumination of an object interior or exterior of a vehicle.
24. To utilize code modulation such as with a pseudo random code to permit the unambiguous monitoring of the vehicle exterior in the presence of other vehicles with the same system.
25. To use a chirp frequency modulation technique to aid in determining the distance to an object interior or exterior of a vehicle.
26. To utilize a correlation pattern modulation in a form of code division modulation for determining the distance of an object interior or exterior of a vehicle.
These and other objects and advantages will become apparent from the following description of the preferred embodiments of the vehicle identification and monitoring system of this invention.
Briefly, in order to achieve at least one of the objects and possibly others, a monitoring arrangement for monitoring an environment exterior of a vehicle comprises at least one receiver arranged to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves, and a processor coupled to the receiver(s) and comprising pattern recognition means for processing the signal to provide a classification, identification and/or location of the exterior object. The pattern recognition means, such as a neural computer or neural network and the like, apply a pattern recognition algorithm generated from data of possible exterior objects and patterns of received waves from the possible exterior objects. In this manner, one or more systems in the vehicle may be coupled to the processor such that the operation the system(s) is/are affected in response to the classification, identification and/or location of the exterior object.
This affected system may be a display viewable by the driver and arranged to show an image or icon of the exterior object or objects. The affected system may also be an internally deployed or externally deployed airbag, deployment of which is controlled based on the identification of the exterior object.
A pair of receivers spaced apart from one another may be used and the receiver may be arranged to receive infrared waves. If a transmitter is provided to transmit waves into the environment exterior of the vehicle, then the receiver is arranged to receive waves transmitted by the transmitter and reflected by any exterior objects.
A measurement system may be provided to measure a distance between the exterior object and the vehicle. The measurement system may be a radar or laser radar system.
Another monitoring arrangement for monitoring an environment exterior of a vehicle in accordance with the invention comprises at least one CCD array positioned to obtain images of the environment exterior of the vehicle and a processor coupled to the CCD array(s) and comprising pattern recognition means for processing the images obtained by the CCD array(s) to provide a classification, identification and/or location of the exterior object. In this manner, one or more systems in the vehicle may be coupled to the processor such that the operation the system(s) is/are affected in response to the classification, identification and/or location of the exterior object. Optionally, the pattern recognition means apply a pattern recognition algorithm generated from data of possible exterior objects and images from the CCD array(s) of the possible exterior objects. The same enhancements described for the arrangement above can be used in this arrangement as well.
In another embodiment of the invention, at least one receiver is arranged on a rear view mirror of the vehicle to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves and a processor is coupled to the receiver(s) and arranged to classify, identify and/or locate the exterior object based on the signal. As such, the operation of one or more systems in the vehicle, coupled to the processor, can be affected in response to the classification, identification and/or location of the exterior object. The same enhancements described for the arrangements above can be used in this embodiment as well.
Instead of being arranged on the rear view mirror, the receiver can be arranged on a B-pillar of the vehicle to receive waves from the environment exterior of and on the side of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves. In addition, the receiver could be arranged in a door window trim panel of the vehicle to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves. Other mounting locations include in a rear window and in a C-pillar of the vehicle.